GH-48864: [C++] Support customizing more Zstd parameters by HuaHuaY · Pull Request #48865 · apache/arrow

HuaHuaY · 2026-01-15T12:52:53Z

Rationale for this change

Support customizing more Zstd parameters to allow users to optimize the performance of parquet page compression/decompression.

What changes are included in this PR?

Add a new class ZstdCodecOptions inherited from CodecOptions.
Create ZSTD_CCtx/ZSTD_DCtx explicitly in ZSTDCodec.

Are these changes tested?

Yes.

Are there any user-facing changes?

Yes. The new class ZstdCodecOptions is added.

GitHub Issue: [C++] Support customizing more Zstd parameters #48864

github-actions · 2026-01-15T12:53:19Z

⚠️ GitHub issue #48864 has been automatically assigned in GitHub to PR creator.

HuaHuaY · 2026-01-15T12:55:37Z

@pitrou @wgtmac @mapleFU Please take a look. This PR only uses context and does not reuse it. I create a new context every time I call Compress and Decompress methods. I guess this is the same behavior as before.

pitrou

Thank you @HuaHuaY . This can be a useful improvement, I've made a bunch of comments below.

cpp/src/arrow/util/compression.h

cpp/src/arrow/util/compression_test.cc

cpp/src/arrow/util/compression_zstd.cc

cpp/src/parquet/column_writer_test.cc

cpp/src/parquet/properties_test.cc

HuaHuaY · 2026-01-15T16:23:55Z

@pitrou Thank you for your detailed review! I've fixed these comments. Please take a look again.

pitrou

This looks mostly good! Just a few nits.

pitrou · 2026-01-15T16:26:22Z

cpp/src/arrow/util/compression_test.cc

-                                                {5, GZipFormat::DEFLATE, -12, false},
-                                                {-992, GZipFormat::GZIP, 15, false}};
-
+template <std::derived_from<arrow::util::CodecOptions> T>


Neat, a concept 😀

cpp/src/arrow/util/compression_test.cc

pitrou · 2026-01-15T16:31:10Z

cpp/src/arrow/util/compression_zstd.cc

+    ARROW_ASSIGN_OR_RAISE(auto cctx, CreateCCtx());
+    size_t ret = ZSTD_compressCCtx(cctx.get(), output_buffer,
+                                   static_cast<size_t>(output_buffer_len), input,
+                                   static_cast<size_t>(input_len), compression_level_);


Funny that the compression level is passed a second time here, after we already gave it to ZSTD_CCtx_setParameter. I suppose it doesn't matter?

Context will also be passed to ZSTDCompressor and it needs compression_level_. We can move auto ret = ZSTD_CCtx_setParameter(cctx.get(), ZSTD_c_compressionLevel, compression_level_) to MakeCompressor though I think the readability will decrease a bit.

After I check the Zstd's document again, I think there is a mistake here. We need to use ZSTD_compress2 instead of ZSTD_compressCCtx because ZSTD_compressCCtx will reset other parameters. Thank you for your good attention!

/*! ZSTD_compressCCtx() : * Same as ZSTD_compress(), using an explicit ZSTD_CCtx. * Important : in order to mirror `ZSTD_compress()` behavior, * this function compresses at the requested compression level, * __ignoring any other advanced parameter__ . * If any advanced parameter was set using the advanced API, * they will all be reset. Only @compressionLevel remains. */ ZSTDLIB_API size_t ZSTD_compressCCtx(ZSTD_CCtx* cctx, void* dst, size_t dstCapacity, const void* src, size_t srcSize, int compressionLevel); /*! ZSTD_compress2() : * Behave the same as ZSTD_compressCCtx(), but compression parameters are set using the advanced API. * (note that this entry point doesn't even expose a compression level parameter). * ZSTD_compress2() always starts a new frame. * Should cctx hold data from a previously unfinished frame, everything about it is forgotten. * - Compression parameters are pushed into CCtx before starting compression, using ZSTD_CCtx_set*() * - The function is always blocking, returns when compression is completed. * NOTE: Providing `dstCapacity >= ZSTD_compressBound(srcSize)` guarantees that zstd will have * enough space to successfully compress the data, though it is possible it fails for other reasons. * @return : compressed size written into `dst` (<= `dstCapacity), * or an error code if it fails (which can be tested using ZSTD_isError()). */ ZSTDLIB_API size_t ZSTD_compress2( ZSTD_CCtx* cctx, void* dst, size_t dstCapacity, const void* src, size_t srcSize);

I have added a new test TEST(TestCodecMisc, ZstdLargerWindowLog) to ensure the other parameters work effectively.

cpp/src/arrow/util/compression_zstd.cc

wgtmac

Looks good to me (I haven't gone through the test). Just left some minor comments.

wgtmac · 2026-01-16T08:14:40Z

cpp/src/parquet/properties_test.cc

 }

 TEST(TestWriterProperties, SetCodecOptions) {
+  constexpr int ZSTD_c_windowLog = 101;


Should we use kZstdCWindowLog to be consistent with this code base?

I suggest to keep the name style of zstd. Because I think it is more readable, although it is inconsistent with our coding style.

I agree we can just keep this in the tests.

cpp/src/arrow/util/compression.h

HuaHuaY · 2026-01-19T09:56:10Z

@pitrou @mapleFU Do you have any suggestions? Please take a look again.

cpp/src/arrow/util/compression_test.cc

cpp/src/arrow/util/compression_zstd.cc

pitrou · 2026-01-19T14:07:44Z

cpp/src/parquet/properties_test.cc

 }

 TEST(TestWriterProperties, SetCodecOptions) {
+  constexpr int ZSTD_c_windowLog = 101;


I agree we can just keep this in the tests.

cpp/src/arrow/util/compression_zstd.cc

mapleFU

General LGTM

pitrou

+1, thank you @HuaHuaY . I'll rebase for CI.

pitrou · 2026-01-21T10:34:40Z

@github-actions crossbow submit -g cpp

github-actions · 2026-01-21T10:38:45Z

Revision: 2af1945

Submitted crossbow builds: ursacomputing/crossbow @ actions-512d34afcc

Task	Status
example-cpp-minimal-build-static
example-cpp-minimal-build-static-system-dependency
example-cpp-tutorial
test-build-cpp-fuzz
test-conda-cpp
test-conda-cpp-valgrind
test-debian-12-cpp-amd64
test-debian-12-cpp-i386
test-debian-experimental-cpp-gcc-15
test-fedora-42-cpp
test-ubuntu-22.04-cpp
test-ubuntu-22.04-cpp-20
test-ubuntu-22.04-cpp-bundled
test-ubuntu-22.04-cpp-emscripten
test-ubuntu-22.04-cpp-no-threading
test-ubuntu-24.04-cpp
test-ubuntu-24.04-cpp-bundled-offline
test-ubuntu-24.04-cpp-gcc-13-bundled
test-ubuntu-24.04-cpp-gcc-14
test-ubuntu-24.04-cpp-minimal-with-formats
test-ubuntu-24.04-cpp-thread-sanitizer

pitrou · 2026-01-21T11:15:29Z

@HuaHuaY The new tests need to be skipped if zstd is not available:
https://github.com/ursacomputing/crossbow/actions/runs/21206351575/job/61003673870#step:6:8214


[ RUN      ] TestCodecMisc.SpecifyCodecOptionsZstd
/arrow/cpp/src/arrow/util/compression_test.cc:458: Skipped
Support for this codec hasn't been built

[  SKIPPED ] TestCodecMisc.SpecifyCodecOptionsZstd (0 ms)
[ RUN      ] TestCodecMisc.ZstdLargerWindowLog
/arrow/cpp/src/arrow/util/compression_test.cc:549: Failure
Failed
'_error_or_value49.status()' failed with NotImplemented: Support for codec 'zstd' not built

[  FAILED  ] TestCodecMisc.ZstdLargerWindowLog (219 ms)
[ RUN      ] TestCodecMisc.ZstdStreamLargerWindowLog
/arrow/cpp/src/arrow/util/compression_test.cc:567: Failure
Failed
'_error_or_value51.status()' failed with NotImplemented: Support for codec 'zstd' not built

[  FAILED  ] TestCodecMisc.ZstdStreamLargerWindowLog (283 ms)

HuaHuaY · 2026-01-21T11:57:39Z

@HuaHuaY The new tests need to be skipped if zstd is not available: https://github.com/ursacomputing/crossbow/actions/runs/21206351575/job/61003673870#step:6:8214


[ RUN      ] TestCodecMisc.SpecifyCodecOptionsZstd
/arrow/cpp/src/arrow/util/compression_test.cc:458: Skipped
Support for this codec hasn't been built

[  SKIPPED ] TestCodecMisc.SpecifyCodecOptionsZstd (0 ms)
[ RUN      ] TestCodecMisc.ZstdLargerWindowLog
/arrow/cpp/src/arrow/util/compression_test.cc:549: Failure
Failed
'_error_or_value49.status()' failed with NotImplemented: Support for codec 'zstd' not built

[  FAILED  ] TestCodecMisc.ZstdLargerWindowLog (219 ms)
[ RUN      ] TestCodecMisc.ZstdStreamLargerWindowLog
/arrow/cpp/src/arrow/util/compression_test.cc:567: Failure
Failed
'_error_or_value51.status()' failed with NotImplemented: Support for codec 'zstd' not built

[  FAILED  ] TestCodecMisc.ZstdStreamLargerWindowLog (283 ms)

My mistake. I have added an if statement to skip the tests. @pitrou

pitrou · 2026-01-21T12:00:10Z

@github-actions crossbow submit cppminimal*

github-actions · 2026-01-21T12:02:25Z

Revision: a6b30fd

Submitted crossbow builds: ursacomputing/crossbow @ actions-3fe0b907c4

Task	Status
example-cpp-minimal-build-static
example-cpp-minimal-build-static-system-dependency
test-ubuntu-24.04-cpp-minimal-with-formats

HuaHuaY · 2026-01-21T12:06:19Z

Sorry that I have refactored a little code. Could you help me take a look and approve this PR again? @pitrou

pitrou · 2026-01-21T12:11:09Z

@github-actions crossbow submit cppminimal*

github-actions · 2026-01-21T12:13:26Z

Revision: d5c8568

Submitted crossbow builds: ursacomputing/crossbow @ actions-c4d50b6264

Task	Status
example-cpp-minimal-build-static
example-cpp-minimal-build-static-system-dependency
test-ubuntu-24.04-cpp-minimal-with-formats

pitrou · 2026-01-21T13:36:12Z

Thanks again @HuaHuaY !

conbench-apache-arrow · 2026-01-21T19:07:26Z

After merging your PR, Conbench analyzed the 3 benchmarking runs that have been run so far on merge-commit 3ed9169.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about 4 possible false positives for unstable benchmarks that are known to sometimes produce them.

HuaHuaY requested a review from wgtmac as a code owner January 15, 2026 12:52

github-actions bot added Component: Parquet Component: C++ awaiting review Awaiting review labels Jan 15, 2026

HuaHuaY force-pushed the fix_issue_48864 branch from e60e703 to 730eb94 Compare January 15, 2026 12:56

pitrou reviewed Jan 15, 2026

View reviewed changes

github-actions bot added awaiting committer review Awaiting committer review awaiting review Awaiting review and removed awaiting review Awaiting review labels Jan 15, 2026

HuaHuaY force-pushed the fix_issue_48864 branch from 730eb94 to 4ab207e Compare January 15, 2026 16:20

github-actions bot added awaiting review Awaiting review and removed awaiting review Awaiting review awaiting committer review Awaiting committer review labels Jan 15, 2026

pitrou reviewed Jan 15, 2026

View reviewed changes

github-actions bot added awaiting committer review Awaiting committer review and removed awaiting review Awaiting review labels Jan 15, 2026

HuaHuaY force-pushed the fix_issue_48864 branch 2 times, most recently from dfbca3b to 84dd650 Compare January 15, 2026 17:36

mapleFU reviewed Jan 16, 2026

View reviewed changes

cpp/src/arrow/util/compression_zstd.cc Show resolved Hide resolved

HuaHuaY force-pushed the fix_issue_48864 branch 2 times, most recently from 5d26717 to 698aefd Compare January 16, 2026 07:58

wgtmac reviewed Jan 16, 2026

View reviewed changes

wgtmac approved these changes Jan 16, 2026

View reviewed changes

pitrou reviewed Jan 19, 2026

View reviewed changes

HuaHuaY force-pushed the fix_issue_48864 branch from 20955d7 to c6c7282 Compare January 19, 2026 14:32

pitrou reviewed Jan 19, 2026

View reviewed changes

cpp/src/arrow/util/compression_zstd.cc Outdated Show resolved Hide resolved

mapleFU reviewed Jan 20, 2026

View reviewed changes

pitrou approved these changes Jan 21, 2026

View reviewed changes

pitrou force-pushed the fix_issue_48864 branch from 4dd93e4 to 2af1945 Compare January 21, 2026 10:34

HuaHuaY added 2 commits January 21, 2026 19:54

support customizing more Zstd parameters

747e123

address comments

28a3998

HuaHuaY force-pushed the fix_issue_48864 branch from 2af1945 to a6b30fd Compare January 21, 2026 11:56

pitrou approved these changes Jan 21, 2026

View reviewed changes

fix tests without zstd

d5c8568

HuaHuaY force-pushed the fix_issue_48864 branch from a6b30fd to d5c8568 Compare January 21, 2026 12:05

pitrou approved these changes Jan 21, 2026

View reviewed changes

pitrou merged commit 3ed9169 into apache:main Jan 21, 2026
52 checks passed

pitrou removed the awaiting committer review Awaiting committer review label Jan 21, 2026

pitrou mentioned this pull request Jan 21, 2026

[C++] Support customizing more Zstd parameters #48864

Closed

HuaHuaY deleted the fix_issue_48864 branch January 21, 2026 14:16

Comments

Conversation

HuaHuaY commented Jan 15, 2026 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

github-actions bot commented Jan 15, 2026

Uh oh!

HuaHuaY commented Jan 15, 2026

Uh oh!

pitrou left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

HuaHuaY commented Jan 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pitrou left a comment

Choose a reason for hiding this comment

Uh oh!

pitrou Jan 15, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

pitrou Jan 15, 2026

Choose a reason for hiding this comment

Uh oh!

HuaHuaY Jan 15, 2026

Choose a reason for hiding this comment

Uh oh!

HuaHuaY Jan 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

HuaHuaY Jan 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

wgtmac left a comment

Choose a reason for hiding this comment

Uh oh!

wgtmac Jan 16, 2026

Choose a reason for hiding this comment

Uh oh!

HuaHuaY Jan 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pitrou Jan 19, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

HuaHuaY commented Jan 19, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pitrou Jan 19, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mapleFU left a comment

Choose a reason for hiding this comment

Uh oh!

pitrou left a comment

HuaHuaY commented Jan 15, 2026 •

edited by github-actions bot

Loading

HuaHuaY commented Jan 15, 2026 •

edited

Loading

HuaHuaY Jan 15, 2026 •

edited

Loading

HuaHuaY Jan 15, 2026 •

edited

Loading

HuaHuaY Jan 16, 2026 •

edited

Loading

HuaHuaY commented Jan 21, 2026 •

edited

Loading

HuaHuaY commented Jan 21, 2026 •

edited

Loading